plausible: init at 1.3.0 #124055

Ma27 · 2021-05-22T18:24:12Z

Motivation for this change

plausible is a simple and privacy-friendly alternative to Google Analytics based on Elixir.

cc @happysalada @lheckemann @kvz

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Added a release notes entry if the change is major or breaking
Fits CONTRIBUTING.md.

r-rmcgibbo · 2021-05-22T18:49:30Z

Result of nixpkgs-review pr 124055 at 9eb694c5 run on aarch64-linux 1

1 package failed to build:

nixos-install-tools

1 package built successfully:

plausible

4 suggestions:

warning: unused-argument

Unused argument: glibcLocales.
Near pkgs/servers/web-apps/plausible/default.nix:1:47:

  |
1 | { lib, stdenv, beamPackages, fetchFromGitHub, glibcLocales, cacert
  |                                               ^

warning: unused-argument
Unused argument: stdenv.
Near pkgs/servers/web-apps/plausible/default.nix:1:8:
```
  |
1 | { lib, stdenv, beamPackages, fetchFromGitHub, glibcLocales, cacert
  |        ^
```

warning: unused-argument

Unused argument: cacert.
Near pkgs/servers/web-apps/plausible/default.nix:1:61:

  |
1 | { lib, stdenv, beamPackages, fetchFromGitHub, glibcLocales, cacert
  |                                                             ^

warning: fixup-phase
fixupPhase should not be overridden, use postFixup instead.

Near pkgs/development/beam-modules/mix-release.nix:77:3:
```
   |
77 |   fixupPhase = ''
   |   ^
```

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

Result of nixpkgs-review pr 124055 at 9eb694c5 run on x86_64-linux 1

1 package failed to build:

nixos-install-tools

1 package built successfully:

plausible

4 suggestions:

warning: unused-argument
Unused argument: stdenv.
Near pkgs/servers/web-apps/plausible/default.nix:1:8:
```
  |
1 | { lib, stdenv, beamPackages, fetchFromGitHub, glibcLocales, cacert
  |        ^
```

warning: unused-argument

Unused argument: glibcLocales.
Near pkgs/servers/web-apps/plausible/default.nix:1:47:

  |
1 | { lib, stdenv, beamPackages, fetchFromGitHub, glibcLocales, cacert
  |                                               ^

warning: fixup-phase
fixupPhase should not be overridden, use postFixup instead.

Near pkgs/development/beam-modules/mix-release.nix:77:3:
```
   |
77 |   fixupPhase = ''
   |   ^
```

warning: unused-argument

Unused argument: cacert.
Near pkgs/servers/web-apps/plausible/default.nix:1:61:

  |
1 | { lib, stdenv, beamPackages, fetchFromGitHub, glibcLocales, cacert
  |                                                             ^

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

nixos/modules/services/web-apps/plausible.nix

pkgs/servers/web-apps/plausible/default.nix

nixos/modules/services/web-apps/plausible.nix

happysalada · 2021-05-23T00:43:43Z

nixos/modules/services/web-apps/plausible.nix

+            ''} plausible";
+          };
+        };
+      }


Would there be a need to add a service for a remote shell ?
In order to be able to connect to a running instance, you need to /bin/plausible remote with the necessary environment variable.
I'm just thinking it might be a very nice convenience in order to debug or check what is happening. It definitely might not be useful for the average user.

Good idea! I guess it's nicer though to have a wrapper-script for that which can be executed from the shell. An example for what I mean would be nextcloud-occ.

It looks good! I'm not sure I understand the way secrets are managed, but otherwise it looks good!

Hmm even if the environment is correctly set up, plausible remote complains about a missing HOME dir even though it's set (checked by looking at /proc/pid/environ)... will take a look at this later.

aanderse

Module LGTM and I approve as is, though as mentioned I think passing a few files to EnvironmentFile instead of the accepting situation might be beneficial... but hey that's just my opinion.

I don't know anything about elixir packaging so maybe someone else can take a look at the package portion of this PR.

nixos/modules/services/web-apps/plausible.nix

Ma27 · 2021-05-23T16:02:22Z

Thanks a lot for the reviews folks! Will work on all that next week :)

nixos/modules/services/web-apps/plausible.nix

lheckemann

Nice work!

nixos/modules/services/web-apps/plausible.nix

pkgs/servers/web-apps/plausible/update.sh

SuperSandro2000 · 2021-05-27T01:23:07Z

nixos/modules/services/web-apps/plausible.xml

@@ -0,0 +1,51 @@
+<chapter xmlns="http://docbook.org/ns/docbook"


Please convert nixos docs to markdown.

nixos/tests/plausible.nix

Needed to modify e.g. a few things in the `plausible` package to get it running properly.

* Most significant is probably the patching necessary to run plausible with postgres without superuser privilege. This change includes: * updating ecto_sql to 3.6 where `CREATE DATABASE` is only executed if it doesn't exist[1]. * patching a migration to only modify the `users.email` column (to use `citext` rather than creating the extension. `plausible-postgres` takes care of that). * Correctly declare dependencies in systemd. * A few minor fixes. [1] elixir-ecto/ecto_sql@051baf6

Co-authored-by: Raphael Megzari <raphael@megzari.com>

happysalada · 2021-06-02T01:43:17Z

nixos/tests/plausible.nix

+      };
+      server = {
+        baseUrl = "http://localhost:8000";
+        secretKeybaseFile = "${pkgs.writeText "dont-try-this-at-home" "nannannannannannannannannannannannannannannannannannannan_batman!"}";


pkgs/servers/web-apps/plausible/default.nix

happysalada

Mostly looks good to me.
A couple of comments added would be great but not an unbendable requirement.

What about the doc issue? I don't know anything about xml vs markdown, I'm willing to go either way, do we want to address the issue in this PR?

Ma27 · 2021-06-02T09:25:11Z

What about the doc issue? I don't know anything about xml vs markdown, I'm willing to go either way, do we want to address the issue in this PR?

AFAICS there isn't really much migrated, so it's not clear to me how I can do the fancy stuff such as referencing options in code-examples or callout-lists in markdown. Probably cheaper to run pandoc once anyways 🤷

I'm not sure you need this patch. If you create the extension before hand, it won't try to create it again.

I think that this wasn't sufficient, but I have a new theory why. Gotta check if I can remove it :)

Turns out this isn't needed if the module correctly adds the `citext` extension to the `plausible` database.

happysalada · 2021-06-02T10:21:41Z

Thanks a lot for doing this!

nh2 · 2021-07-12T20:52:44Z

Hi, I'm currently trying this out.

How is one supposed to set secretKeybaseFile in production?

Because DynamicUser is used, I cannot know the user ID ahead of time that will need to read that file, and I certainly don't want to put it world-readable in the nix store.

Are there any pointers I could read about how one does that?

lheckemann · 2021-07-12T20:56:02Z

Off the top of my head, I think LoadCredential= (see systemd.exec(5)) could be helpful for this, but I'm not sure if applying it here is trivial or not.

Ma27 · 2021-07-12T21:07:09Z

I'm not sure right now, but IIRC it was sufficient to set e.g. SupplementaryGroups and make the secret readable by some sort of dedicated group (which is also the default approach used by e.g. deployment.keys of nixops or sops-nix). If you're having problems with this, I can check e.g. tomorrow :)

nh2 · 2021-07-12T21:32:47Z

nixos/modules/services/web-apps/plausible.nix

+
+  # FIXME consider using LoadCredential as soon as it actually works.
+  envSecrets = ''
+    export ADMIN_USER_PWD="$(<${cfg.adminUser.passwordFile})"


As pretty much 100% of shell scripts, this is probably wrong:

The shell this runs in doesn't have set -e nor any other reasonable way of error propagation, so when the configured file isn't readable by the DynamicUser, this code just prints e.g.

/nix/store/rwfwdk34fyz9qqyvn4vs0lnswscd73x0-plausible-setup: line 2: /var/lib/plausible/plausible-admin-user-password-file: Permission denied /nix/store/rwfwdk34fyz9qqyvn4vs0lnswscd73x0-plausible-setup: line 3: /var/lib/plausible/plausible-secret-keybase-file: Permission denied

and then happily goes on with its day, with ADMIN_USER_PWD and SECRET_KEY_BASE set to the empty string.

Even if it had set -e, it would still be wrong, in the same way as the scripts of Elixir itself here:

elixir-lang/elixir#11114

Fixes in PR #130062

nh2 · 2021-07-13T03:30:35Z

nixos/modules/services/web-apps/plausible.nix

+                  psql -d plausible <<< "UPDATE users SET email_verified=true;"
+                fi
+              ''}
+            ''} plausible-setup";


What is the argument to this shell script for?

It calls e.g.

/nix/store/skpa6a2l84ydidgxnpv63fnc1fsxyv6k-plausible-setup plausible-setup

But the shell script defined directly above does not use this argument.

To quote systemd.service(5):

If the executable path is prefixed with "@", the second specified
token will be passed as "argv[0]" to the executed process (instead of
the actual filename), followed by the further arguments specified.

A similar approach for shorter names is done in e.g. the module of Hydra or the firewall :)

Does that answer your question?

I actually wanted to ask you why you used @ in this case? (rather without)
You don't actually use the argument in this case right?

@Ma27 Ah yes, thank you! Is the intent to make it look better in ps/htop?

nh2 · 2021-07-13T03:31:01Z

nixos/modules/services/web-apps/plausible.nix

+            ExecStart = "@${pkgs.writeShellScript "plausible" ''
+              ${envSecrets}
+              plausible start
+            ''} plausible";


Same question as https://github.com/NixOS/nixpkgs/pull/124055/files#r668400359

happysalada · 2021-07-13T08:02:09Z

The "best" way to do this would be to use LoadCredentials.
The current problem is that systemd is blocked at version 147 and LoadCredentials doesn't work with that version.
The other problem is to make upstream be able to read a configuration variable from a file. I've worked with upstream for that change, and it should be available in the next release.

So in summary no perfect way to do this. I haven't found a way without LoadCredentials to not have your variable readable.

nh2 · 2021-07-13T12:44:27Z

@happysalada Another problem I have with DynamicUser is that I cannot figure out how to debug things well with it.

For me the plausible startup fails with

** (RuntimeError) Code: 81, e.displayText() = DB::Exception: Database `` doesn't exist (version 20.5.2.7)

because in some of its calls from createdb.sh, it seems to not pass the CLICKHOUSE_DATABASE_URL=http://127.0.0.1:8123/default setting all the way through, and make ?database= requests (with empty string as the DB name), right after making two ?database=default requests.

The usual approach is to sudo -u plausible /path/to/createdb.sh, running it directly, perhaps with strace.

This seems impossible with DynamicUser, because there is no user you could sudo -u as.

So while I like the concept of DynamicUser, it seems to currently block these more fundamental debugging / sysadmin facilities.

Ma27 · 2021-07-13T12:52:03Z

You should be able to do something like systemd-run -p DynamicUser=yes -p StateDirectory=plausible -- your cmd. If a StateDirectory is used, the DynamicUser will be named after and then you have an environment similar to the service itself.

happysalada · 2021-07-13T13:12:50Z

I love this conversation!
I originally had some problems with DynamicUser for the backup. I ended up creating another User for backup and granting all priviledges on db for it.
I don't really like having to create another user and granting all priviledges. Is there some other black magic command from systemd that has something for that?
Or do you guys also have another user for backups on production?
(I use a systemd-timer, with a simple shell script that basically dumps the db, gathers other files like uploads and the likes and borgbackup it to a remote)

nh2 · 2021-07-14T03:38:24Z

pkgs/servers/web-apps/plausible/default.nix

+
+    # We need ecto 3.6 as this version checks whether the database exists before
+    # trying to create it. The creation attempt would always require super-user privileges
+    # and since 3.6 this isn't the case anymore.


This patch breaks plausible for me:

nixos/plausible: Fix shell scripting errors, runtime fixes #130062 (comment)

1656e8e

github-actions bot added 6.topic: nixos 8.has: module (update) labels May 22, 2021

ofborg bot added 8.has: package (new) 11.by: package-maintainer 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels May 22, 2021

happysalada reviewed May 23, 2021

View reviewed changes

nixos/modules/services/web-apps/plausible.nix Outdated Show resolved Hide resolved

happysalada reviewed May 23, 2021

View reviewed changes

nixos/modules/services/web-apps/plausible.nix Show resolved Hide resolved

happysalada reviewed May 23, 2021

View reviewed changes

pkgs/servers/web-apps/plausible/default.nix Outdated Show resolved Hide resolved

happysalada reviewed May 23, 2021

View reviewed changes

pkgs/servers/web-apps/plausible/default.nix Show resolved Hide resolved

happysalada reviewed May 23, 2021

View reviewed changes

pkgs/servers/web-apps/plausible/default.nix Outdated Show resolved Hide resolved

happysalada reviewed May 23, 2021

View reviewed changes

nixos/modules/services/web-apps/plausible.nix Show resolved Hide resolved

happysalada reviewed May 23, 2021

View reviewed changes

aanderse reviewed May 23, 2021

View reviewed changes

nixos/modules/services/web-apps/plausible.nix Show resolved Hide resolved

nixos/modules/services/web-apps/plausible.nix Outdated Show resolved Hide resolved

nixos/modules/services/web-apps/plausible.nix Show resolved Hide resolved

happysalada reviewed May 24, 2021

View reviewed changes

nixos/modules/services/web-apps/plausible.nix Outdated Show resolved Hide resolved

happysalada reviewed May 24, 2021

View reviewed changes

nixos/modules/services/web-apps/plausible.nix Outdated Show resolved Hide resolved

happysalada reviewed May 24, 2021

View reviewed changes

nixos/modules/services/web-apps/plausible.nix Show resolved Hide resolved

lheckemann reviewed May 26, 2021

View reviewed changes

nixos/modules/services/web-apps/plausible.nix Show resolved Hide resolved

lheckemann suggested changes May 26, 2021

View reviewed changes

SuperSandro2000 reviewed May 27, 2021

View reviewed changes

Ma27 added 2 commits May 28, 2021 23:29

yarn2nix: set name to ${pname}-${version} by default

7cd86c7

plausible: init at 1.3.0

34041e4

SuperSandro2000 reviewed May 28, 2021

View reviewed changes

nixos/tests/plausible.nix Outdated Show resolved Hide resolved

Ma27 added 2 commits May 29, 2021 16:12

beam/fetchMixDeps: allow adding patches

50bc40e

Needed to modify e.g. a few things in the `plausible` package to get it running properly.

Ma27 force-pushed the plausible branch from 9eb694c to 68c8a43 Compare May 29, 2021 14:16

github-actions bot added the 6.topic: erlang label May 29, 2021

plausiblew: cleanup build & update script

85c7a4a

Ma27 and others added 2 commits May 31, 2021 19:01

plausible: correctly run digest task

6e12c74

plausible: platforms.{linux -> unix}

4d32493

Co-authored-by: Raphael Megzari <raphael@megzari.com>

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 and removed 10.rebuild-darwin: 0 labels May 31, 2021

plausible: minor polishing

885512d

Ma27 requested review from happysalada and aanderse June 1, 2021 16:05

happysalada reviewed Jun 2, 2021

View reviewed changes

pkgs/servers/web-apps/plausible/default.nix Outdated Show resolved Hide resolved

happysalada reviewed Jun 2, 2021

View reviewed changes

pkgs/servers/web-apps/plausible/default.nix Show resolved Hide resolved

happysalada reviewed Jun 2, 2021

View reviewed changes

plausible: remove create extension patch

204d1b6

Turns out this isn't needed if the module correctly adds the `citext` extension to the `plausible` database.

Ma27 requested a review from happysalada June 2, 2021 09:48

happysalada merged commit c4afcbb into NixOS:master Jun 2, 2021

Ma27 deleted the plausible branch June 2, 2021 10:36

nh2 reviewed Jul 12, 2021

View reviewed changes

nh2 reviewed Jul 13, 2021

View reviewed changes

nh2 mentioned this pull request Jul 14, 2021

nixos/plausible: Fix shell scripting errors, runtime fixes #130062

Merged

11 tasks

nh2 reviewed Jul 14, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

plausible: init at 1.3.0 #124055

plausible: init at 1.3.0 #124055

Ma27 commented May 22, 2021

r-rmcgibbo commented May 22, 2021 •

edited

Loading

happysalada May 23, 2021

Ma27 May 23, 2021

happysalada May 24, 2021

Ma27 May 29, 2021

aanderse left a comment

Ma27 commented May 23, 2021

lheckemann left a comment

SuperSandro2000 May 27, 2021

happysalada Jun 2, 2021

happysalada left a comment

Ma27 commented Jun 2, 2021

happysalada commented Jun 2, 2021

nh2 commented Jul 12, 2021

lheckemann commented Jul 12, 2021

Ma27 commented Jul 12, 2021

nh2 Jul 12, 2021

nh2 Jul 13, 2021

nh2 Jul 13, 2021

nh2 Jul 13, 2021

Ma27 Jul 13, 2021

happysalada Jul 13, 2021

nh2 Jul 13, 2021

nh2 Jul 13, 2021

happysalada commented Jul 13, 2021

nh2 commented Jul 13, 2021

Ma27 commented Jul 13, 2021

happysalada commented Jul 13, 2021

nh2 Jul 14, 2021

		@@ -0,0 +1,51 @@
		<chapter xmlns="http://docbook.org/ns/docbook"

plausible: init at 1.3.0 #124055

plausible: init at 1.3.0 #124055

Conversation

Ma27 commented May 22, 2021

Motivation for this change

Things done

r-rmcgibbo commented May 22, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aanderse left a comment

Choose a reason for hiding this comment

Ma27 commented May 23, 2021

lheckemann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

happysalada left a comment

Choose a reason for hiding this comment

Ma27 commented Jun 2, 2021

happysalada commented Jun 2, 2021

nh2 commented Jul 12, 2021

lheckemann commented Jul 12, 2021

Ma27 commented Jul 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

happysalada commented Jul 13, 2021

nh2 commented Jul 13, 2021

Ma27 commented Jul 13, 2021

happysalada commented Jul 13, 2021

Choose a reason for hiding this comment

r-rmcgibbo commented May 22, 2021 •

edited

Loading